Indonesian Journal of Electrical Engineering and Computer Science 
Vol. 25, No. 2, February 2022, pp. 995~1002 
ISSN: 2502-4752, DOI: 10.11591/ijeecs.v25.i2.pp995-1002 o 995 


Internet of things and multi-class deep feature-fusion based 


classification of tomato leaf disease 


Rina Mahakud!, Binod Kumar Pattanayak!, Bibudhendu Pati? 


‘Department of Computer Science and Engineering, Siksha 'O' Anusandhan Deemed to be University 


Institute of Technical Education and Research, Bhubaneswar, India 


*Department of Computer Science, Rama Devi Women’s University, Bhubaneswar, India 


Article Info 


ABSTRACT 


Article history: 


Received Sep 17, 2021 
Revised Dec 20, 2021 
Accepted Dec 24, 2021 


Keywords: 


AlexNet 

CNNs 

SVM 

Tomato leaf disease 
identification 
Transfer learning 


A deep transfer learning (deep-TL) classification model has been proposed 
to diagnose tomato leaf disease. The main challenge of inaccurate 
classification of a convolution neural network (CNN) model was the 
availability of the small-sized dataset. This model deals with the challenges 
like availability of small-sized and imbalanced datasets. The proposed Alex 
support vector machine (SVM) fused hybrid classification (ASFHC) model 
is based on fully fusion technology that avoids overfitting to classify the type 
of disease in tomato leaves. The proposed model achieves the best 
performance in terms of accuracy by data augmentation of the training data. 
It uses a pre-trained network for feature extraction with the modification of 
architecture by concatenating two layers FC6 and FC7 (fully connected 
layer), plus a linear SVM classifier for classification of the disease. The 
uniqueness of the research is although the dataset is not balanced, the 
performance of the model has achieved the maximum. Compared with VGG 
16 and VGG 19, the proposed model (ASFHC) has been evaluated using 


different measuring parameters, indicating remarkable computation time for 
implementation in the internet of things (IoT) domain. The overall accuracy 
attained by the model is 99.62%. 
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1. INTRODUCTION 

Tomato is a useful vegetable that is rich in vitamin C, potassium, folate, and vitamin K. This plant 
has some medicinal values. It is advisable to consume in order to reduce the risk of heart disease and cancer. 
In 2020, world processing tomato council (WPTC) world estimate of tomato production is around 
39.2 million metric tons, mainly due to the increase of the Chinese forecast to 5.6 million tones [1]. The 
growth cycle of the tomato plant is relatively short, that is about three to five months with an average 
temperature 10 to 20 degrees [2]. It is very difficult for a farmer to identify the diseases in tomato plants from 
their past experience because of climate change. In the manual case of cultivation, the process is labor- 
intensive, cost-effective and time-consuming. So, they need domain expert visit to the field, which is costly. 
It is challenging to protect healthy plants from diseases to provide quality and quantitative products in the 
market. Therefore, if diseases are appropriately diagnosed, then precautions can be taken. To replace the 
expert’s advice in traditional method of cultivation, it is very much of necessary to automate the disease 
identification process at very early stage [3]. The type of leaf disease classification by visual observation is 
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prone to error due to environmental stimuli such as shadows, reflections of sunshine, fog, and high humidity [4]. 
Recently advancement in computational tools such as image processing, remote sensing, artificial 
intelligence and deep learning helped to achieve successful application. In last 10 years, deep learning (DL) 
had been applied in various fields like video classification, object detection, and human action recognition. 
Freshly DL has also been successfully applied in agricultural sector for disease identification and prediction 
of disease [5], [6]. In the past decade, the classification algorithm such as support vector machine (SVM) [7], 
k-nearest neighbor (KNN) [8], artificial neural network (ANN) [9], particle swarm optimization [10], random 
forest (RF) [11] and so on were used for plant disease identification. SVM is a popular supervised machine 
learning classifier used for solving both binary and multi class classification problem. Thus, pre-trained 
convolution neural network (CNN) network with SVM classifier improves the performance of the model. 

The main work of this paper is revolved around the vegetable tomato. This is a vegetable which can 
be cultivated throughout the year. The climate changing factors that are humidity, heavy rain, global 
radiation, and temperature. That can affect the growth of the crop [12] and due to these factors’ plants got 
affected by the diseases. So, there are a lot of research work undertaken by the researcher for leaf diseases 
identification of the tomato plant using various models. A convolution neural network (CNN) model was 
proposed in [13] for the tomato plant's leaf-based classification. In this CNN Model, 1400 leaf images had 
been trained in suggested 8 layers and 98.4% of accuracy was obtained. Verma et al. [14] developed a CNN 
architectures-based model which works in two different modes for extracting the features and then the 
extracted feature set was trained in a multiclass SVM to get the final output. The author compared the 
accuracy of the proposed AlexNet model with other two networks SqueezeNet and Inception V3. The 
AlexNet resulting the highest accuracy in transfer learning and feature extraction approaches 89.69% and 
93.4%, respectively for tomato late blight disease. Ashqar and Abu-Naser [15] presented a full-color neural 
network model to extract features in two sub sections. The first section of the model is feature extraction 
which consists of 4 layers of convolutional layers, rectified linear unit (ReLu) and max pooling layer. The 
second section of the model is flattened layer by using two dense layers for both approaches. Luna et al. [16] 
developed a pre-trained deep learning AlexNet architecture to identify the three tomato leaf diseases. The 
author utilizes the model AlexNet both anomaly detection and disease detection. The proposed model had an 
accuracy of 91.6 percent based on 36 samples. Ferentinos [17] developed an optimal deep learning model and 
used dataset contains 87,848 infected images and healthy leaves out of 25 plant species from open datasets. 
The author concluded that the best performance given by VGG convolutional neural network with 99.53% of 
success rate in the classification in comparison to AlexNet one weird trick batch (AlexNetOWTBn) with 
99.49% of success rate. Hassanien et al. [18] proposed a mathematical model moth-flame optimization 
(MFO) and rough set (MFORSFS) approach to automatically detect powdery mildew and early blight of 
tomato disease. Textural pattern features are extracted by using Gabor transform of diseased tomato leaves. 
Here author represented a comparison between particle swarm optimization and genetic algorithms with 
rough sets from which rough set provide better results. MFORS-based classification offers a 91.5% result as 
compared to maximum relevance minimum redundancy (mRMR) based method. 

A Transfer learning approach is proposed in this present work to identify and classify the type of 
disease in real time. For implementation of the model a pretrained network is used with the modification of 
architecture. The prosed fused hybrid model is a classification model based on deep CNN model. The main 
contributions of this work are follows: 

— A fused hybrid model is proposed, which can provide better accuracy with faster execution time withan 
imbalanced dataset by using field images as well as standard dataset to achieve a useful detection of 
tomato plant disease. 

— A Pre-trained transfer learning architecture based on fusion concept is used to analyze tomato leaf disease 
characteristics, with the modification of concatenating two layers FC6 and FC7 (fully connected layer). 


2. MATERIALS AND METHOD 

The classification of the disease identification in tomato plant is shown in Figure 1. In this section, 
detail of image dataset creation from internet of things (IoT) module and implementation of suggested 
classification of the disease models are elaborated. The proposed model is broadly divided into three units 
such as data acquisition (IoT modules sensed the image data), processing unit (using CNN) and classification 
unit are described in subsequent subsection. 


2.1. Image dataset 

Sensors are placed in the field to capture images at regular intervals during crop growth in order to 
monitor for infection. This proposed model is comprised of a device known as the Rasberry PI, which is a 
quad-core 64-bit advanced reduced instruction set computing (RISC) machines (ARM) processor running at 
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1.2 GHz. The web camera is linked to the processor device. Here, a digital camera, web camera, or closed- 
circuit television (CCTV) can be linked to the processor to continuously capture images of leaves and 
generate a database of the leaves. All the modules are connected through IoT network. But in the proposed 
IoT based disease identification scheme, the dataset is collected from ‘Plant village’ database. The dataset 
contains a total of 6660 images, out of which 6578 images are diseased tomato leaf images caused by yellow 
leaf curl virus and mosaic virus, and 82 of which are healthy tomato leaf images. All the images are pre- 
processed for resized to 227x227x3. The dataset contains tomato leaf images which are grouped into three 
classes as tomato mosaic virus, tomato yellow leaf curl, and tomato healthy is shown in Figure 2. These three 
classes are shown in Figures 2(a)-(c) respectively. We had taken 80% and 20% of the samples for training 
and testing purpose respectively. The samples are chosen by the model randomly for each execution. 
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loT Module 
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Figure 1. Block diagram of proposed model for identification of tomato leaf diseases 


(a) (b) (c) 


Figure 2. Sample images of tomato leaf: (a) tomato mosaic virus, (b) tomato yellow leaf curl, and 
(c) tomato healthy 


2.2. Data augmentation 

Data augmentation is a manipulating technique which is done by simple rotation of images (i.e., 90 
degrees right and left rotation and 180 degrees rotation) and flipping operation (i.e., vertical flipping and 
horizontal flipping) to overcome the problem of overfitting, we applied data augmentation. By augmentation, 
the number of sample images are increases to six times to the number of images to the applied augmentation 
images. As a result, the chance to learn the appropriate feature is increased for the network [19]. Table 1 
presents the detail of each class of tomato leaf disease used during the experimentation. 


Table 1. Details of image dataset for tomato leaf disease used in the study 


Leaf diseases Number of images Number of images used Number of images Number of images 
(original) for (augmentation) used for (Training) used for (Testing) 
Tomato mosaic virus 1110 910 5460 200 
(ToMV) 
Tomato yellow leaf curl 5468 5268 31608 200 
virus (TYLCV) 
Total 6578 6178 37068 400 
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3. METHOD 

Generally, transfer learning is the approach of deep learning, which is used for deep feature 
extraction. In this model, we include three pre-trained model for feature extraction. In this work, the feature 
extraction is done through concatenating two fully connected layers (FCL) and from the fully connected 
layers of (FC6 and FC7), extracted features are feed to the linear SVM classifier for classify the disease 
during training the model. 

Process involves in the feature extraction the deep learning approach applied in this model, involves 
in two process that are i) feature extraction and ii) classification. Feature extraction processes can be carried 
out with several convolution layers added with max-pooling layers and output of convolution layers are 
connected to ReLu [20] as a non-linear activation function. The convolution layer involves in the process can 
be learned in supervised or unsupervised manner contains a set of filters. It applies the convolution operation 
on the input image, then the result will be applied to the next layer. In convolution layer, it is calculated by 
using the formula as given in (1). 

x= met +1 (1) 
Where, n=image size having height and width, p is zero padding, d is the filter size and s is the size of the 
stride. 

The pooling layer is used to reduce the dimensions of the feature map due to which it summarizes 
the features generated by convolution layers to makes the model more robust. Generally, two types of 
pooling are taken in to the consideration that are average and max pooling. In case of average polling, the 
average elements are calculated within the receptive field to send the output array. But in max pooling 
maximum number of elements selected from the region in the feature map. Thus, max pooling provides most 
prominent features in the region. 

In this work, we propose transfer learning approach to extract the deep features that are the abstract 
features. From the related case studies of a methodology based on deep learning Ding et al. [19] focused on 
the feature extraction and classification. During experimentation, it was found that the fully connected layer 
FC6 is providing higher accuracy in classification than FC7. Sethy et al. [21], described an in-depth feature- 
based leaf disease identification model. From there experimental study, the authors concluded that FC6 and 
FC7 give better result as compared to FC8. In the proposed model, we have taken those deep features and do 
the concatenation (i.e., concatenation for FC6 and FC7). So, here in this work the deep features extracted 
from FC6 and FC7is considered. From FC6 4096 number of features are extracted and similarly from FC7 
also 4096 number of features are extracted. Then the extracted features from FC6 and FC7 are fused together 
to enhance the dimension of features. The features fusion process is done by concatenation the two features 
i.e, FC6 and FC7 to form a high dimensional feature. Here the dimension of the features is added to provide 
4096 + 4096 = 8192. With consideration of this number of features the liner SVM classifying three 
varieties of tomato disease such as tomato mosaic virus (TOMV), tomato yellow leaf curl virus (TYLCV), 
and tomato healthy. 

Said et al. [22] suggested for image classification in machine learning most preferable classifier is 
SVM. This classifier separates the datapoint using a line of hyperplane for dataset labelling during the 
training process for tomato leaf disease. In this paper linear-SVM is used for disease classification based on 
deep features of pre-trained model Alexnet. “To train the SVM, the function ‘fit class error-correcting output 
codes (fitcecoc)' was used. This function returns full trained multiclass error-correcting output of the model. 
The function ‘fitcecoc’ uses K(K-1)/2, binary SVM model, using one-vs-all coding design. Here, K is a 
unique class label. Because of error correcting output codes and one-vs-all coding design of SVM, the 
performance of classification models was enhanced [21].” The proposed model involved the following steps 
are given in Algorithm 1. 


Algorithm 1: 

Step-I: Input: Diseased leaf images. 

Step-II: Pre-processing: Resize Images to 227x227x3 
dimensions 

Step-III: Image augmentation 

Step-IV: Concatenate FC6 and FC7 layers of the CNN model. 

Step-V: Classification using linear SVM 


4. RESULTS AND DISCUSSION 
In this research model, CNN based transfer learning is used to identify tomato leaf disease with 
giving important to the efficiency of the model in classification and execution time taken by the model. 
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MATLAB 2020 a with add on as deep leaning is used for experimental study with the system having, 
configuration Intel i7 9 generation processor, 16 GB DDR4 RAM, 250 GB SSD & 4TB HDD, NVIDA 
1050Ti 4GB DDR5 GPU with NVIDA cuda_11.1.0_456.43_win10. The classification model's performance 
is evaluated through accuracy. To get maximum accuracy several iterations to be performed and some 
training time will be required for the performance. In this transfer learning approach, the experiment is 
carried out in two phases i.e. training phase and testing phase. 

From subsection 3.1, it was noticed that proposed model performs well in the FC6 and FC7 layer for 
classification. From this observation we included those deep features and concatenate these fully connected 
layers to perform better results. In this section, we experimented a transfer learning pre-trained models based 
on CNN. The hyper parameters used in transfer learning methods are stochastic gradient descent (SGD), 
0.001 for the initial learning rate, 50 for the number of epochs, 0.9 for momentum, 0.2 for drop out, and 16 
for the minibatch scale. The minibatch size can be 8, 16, 32, 64, 128 or 256, depending on the GPU's ability. 
The simulated result of proposed (ASFHC) model, VGG 16, VGG 19 classification model’s is represented by 
confusion matrix given in Figure 3. Figure 3(a) shows the better classification result in camparision to 
Figures 3(b) and (c). 
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Figure 3. Confusion matrix for the (a) ASFHC, (b) VGG 16, and (c) VGG 19 


In this above confusion matrix, each rows represent the actual class and each column represent the 
prediction class whereas diagonal values represent each leaf classes correctly calculated shown in green color 
other values shown in pink color misclassified. In the proposed ASFHC model, first row the confusion matrix 
contains total 1094 instants out of which 1092 instants correctly classified as TYLCV and only 2 instants 
were wrongly classified as ToMV is shown as error. In second row 219 correctly classified as ToMV and 3 
instants were misclassified as TYLCV. In third row all 16 instants are classified as healthy. Thus, the 
performance measures of tomato disease plant are given in Table 2. 


Table 2. Measurement parameters of CNN models with deep features 


Network ASFHC (proposed model) VGG16 VGG19 

Disease Class-1 Class-2 Class-3 Class-1 Class-2 Class-3 Class-1 Class-2 Class-3 
Accuracy of 0.99817 0.98649 1 0.99909 0.95946 1 1 0.95946 1 
single 
Error of 0.0018282 0.013514 0 0.0009141 0.040541 0 0 0.040541 0 
single 


Accuracy in 0.81982 0.16441 0.012012 0.82057 0.15991 0.012012 0.82132 0.15991 0.012012 
total 


Error in 0.0022523 0.0015015 0 0.0067568 0.0007508 0 0.0067568 0 0 
total 

Sensitivity 0.99817 0.98649 1 0.99909 0.95946 1 1 0.95946 1 
Specificity 0.98739 0.9982 1 0.96218 0.9991 1 0.96218 1 1 
Precision 0.99726 0.99095 1 0.99183 0.99533 1 0.99184 1 1 
False 0.012605 0.0018018 0 0.037815 — 0.0009009 0 0.037815 0 0 
positive rate 

Fl1_score 0.99772 0.98871 1 0.99545 0.97706 1 0.9959 0.97931 1 


*Tomato_mosaic_virus=> class1, Tomato_Yellow_Leaf_Curl_Virus=>class-2, Tomato_healthy=>class-3 
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From Table 2 the performance measuring parameters of each disease including healthy tomato plant 
was indicated. The proposed approach not only identified the disease but also distinguishes the specific types 
of diseases. The proposed model provides overall performance is given in Table 3. 


Table 3. Overall performance of the proposed model 


Network Accuracy Error Sensitivity Specificity Precision FPR F1_score Pxecunon 

time (sec) 
ASFHC (proposed model) 0.9962 0.0038 0.9949 0.9952 0.9961 0.0048 0.9955 46 
VGG16+ Linear SVM 0.9925 0.0075 0.9862 0.9871 0.9957 0.0129 0.9908 175 
VGG19+ Linear SVM 0.9932 0.0068 0.9865 0.9874 0.9973 0.0126 0.9917 237 


From Table 3, it was observed that by using fused features of FC6 and FC7, the ASFHC achieves 
highest accuracy which is indicates in bold. All the measuring parameters are obtained the maximum value. 
But as this model is implemented in IoT domain, the response of the model should be faster. On the basis of 
execution time, it was observed that ASFHC model provides the result in just 46sec. whereas VGG 16 takes 
175 sec and VGG 19 takes 237sec. Hence, the ASFHC model with linear SVM is the best classifier model 
can be considered in IoT domain. The proposed model compares the accuracy with existing model are given 
in Table 4. 

From the Table 4 it can be summarized that the plant village dataset was used by most of the 
authors. It contains thousands of tomato leaf images. Most of the above research approach includes SVM 
classifier to detect the diseases. But only Sethy et al. [21] able to provide computation time as 48.65 sec. The 
proposed model i.e., Alex SVM fused hybrid classification (ASFHC) with fused features (FC6 & FC7) and 
linear SVM is the chosen classifier to classify and identify the tomato leaf disease with an accuracy of 
99.62% in computation time 46 sec which can be implemented IoT application domain. In this proposed 
model may have limitations, that it will provides more accurate result in color images rather than gray scale 
images because this model implemented in IoT domain which collect the real time images at normal 
environmental condition. 


Table 4. Comparison of various methods with proposed model 
Authors Dataset Methodology Accuracy Computation time (Sec) 
Verma et al. [14] Plant village dataset CNN architecture (i.e., 93.4% 
transfer learning and 
feature extraction) 
+multiclass SVM 


Sethy et al. [21] http://bcch.ahnw.gov.cn/Right. | Deep Learning +SVM 97.62% 
aspx 48.65 

Brahimi et al. [23] Plant village dataset Pre-processing+ feature 98.660% Not Specified but want 
extraction +SVM or to reduce computation 
Random Forest time 

Durmus et al. [24] Plant village dataset Comparative study of 95.65% 
AlexNet and -- 
SqueezeNet 

Hong et al. [25] Plant village dataset Deep CNN model 97.10% -- 

Altuntas et al. [26] Plant village dataset Deep CNN model 96.99 % 

ASFHC (Proposed Model) Plant village dataset Deep CNN+linear 99.62% 46 
SVM 


5. CONCLUSION 

This proposed model focuses on deep-CNN transfer learning approach to identify tomato leaf 
disease with high accuracy of 99.62%. In this current research, we evaluate the performance of the model 
with linear SVM. The main aim of this research work is to automate the disease identification as it is 
implemented in IoT domain. Here, we are avoiding overfitting by concatenating FC6 and FC7 of the 
architecture due to which all the features are getting concatenated to provide best execution time just at 46 
sec. These network models are evaluated with other existing models out of which ASFHC model performs 
best execution time as compared to other models, which fits to be implemented in IoT domain. This research 
can be further carried forward with a comparative analysis with other existing methods with more variety of 
tomato diseases to achieve better performance. 
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